FIX: Adaptive Scenario bugs#2152
Merged
Merged
Conversation
Resume re-drew an unseeded random.sample before restricting to persisted objective hashes, so whenever max_dataset_size < len(dataset) the fresh draw diverged from the persisted subset and resume aborted with 'persisted objective hash(es) are no longer present in the dataset'. Thread an apply_sampling keyword through the seed-resolution path so resume resolves the full deterministic dataset and lets the persisted hashes reconstruct the exact original subset. Fresh runs still sample as before. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
_generate_next_message_async built the request message with Message.from_prompt, which leaves conversation_id unset (None), then passed that None to set_system_prompt. set_system_prompt calls get_conversation_messages, whose conversation_id filter is skipped when the id is falsy, so it returned every message piece in memory and group_conversation_message_pieces_by_sequence raised 'All message pieces must be from the same conversation' as soon as memory held more than one conversation. This surfaced intermittently in adaptive single_turn runs (context_compliance) once the epsilon-greedy selector picked the simulated path and multiple conversations existed. Scope the system prompt and the generated request message to a fresh unique conversation id so the filter is always applied. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
jsong468
reviewed
Jul 9, 2026
jsong468
approved these changes
Jul 9, 2026
romanlutz
approved these changes
Jul 9, 2026
A falsy conversation_id caused get_message_pieces to skip its filter and silently return pieces from every conversation, which surfaced as the flaky 'same conversation' error. Raise ValueError instead. Two TAP tests were passing only via this footgun (querying an empty conversation_id); retarget them to resolve the real node conversation from memory. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes two pre-existing bugs in the adaptive-scenario resume/attack machinery (surfaced during Phase F, not caused by it).
Bug 1 — Non-deterministic resume under
max_dataset_sizeOn resume,
DatasetAttackConfigurationre-drew an unseededrandom.samplebefore restricting to the persistedobjective_hashes. Whenevermax_dataset_size < len(dataset), the fresh draw diverged from the persisted subset, the intersection dropped them, and resume aborted with "persisted objective hash(es) are no longer present in the dataset."Fix: thread an
apply_samplingkeyword through the seed-resolution path so resume resolves the full deterministic dataset and lets the persisted hashes reconstruct the exact original subset. Fresh runs still sample as before.Bug 2 — Flaky "All message pieces must be from the same conversation"
_generate_next_message_async(simulated conversation) built its request viaMessage.from_prompt, which leavesconversation_idunset (None), then passed thatNonetoset_system_prompt.get_message_piecesskips its conversation filter when the id is falsy, so it returned every piece in memory and the grouper raised as soon as memory held >1 conversation. Intermittent because it needed both multiple conversations in memory and the epsilon-greedy selector to pick the simulatedsingle_turn(context_compliance) path.Fix: scope the system prompt and generated request message to a fresh unique
conversation_idso the filter is always applied.Testing
single_turnrun: failed on first attempt before the fix, 5/5 clean after.tests/unit/scenario/(795) +tests/unit/executor/attack/{multi_turn,component}/(468) pass; pre-commit green.